You know, I've been giving talks on Python and its scientific ecosystem for about three years now... And I always write this bit there, that "Almost" word in italics before my background. You may reasonably wonder now what the heck I've been doing all these years to always introduce myself as an "almost" Aerospace Engineer, right? Well, I promise that I'm taking the required steps to graduate not later than this Autumn, but anyway this talk reflects one of the severe pains I've been going through while carrying my final project.
Let's begin with some questions:
setup.py... without copying a working one from the Internet?If you’re missing a library or program, and that library or program happens to be written in C, you either need root to install it from your package manager, or you will descend into a lovecraftian nightmare of attempted local builds from which there is no escape. You say you need lxml on shared hosting and they don’t have libxml2 installed? Well, fuck you.
"It's easy to build a VM if you automate the install process, and providing that install script for even one OS can demystify the install process for others; conversely, just because you provide a VM doesn't mean that anyone other than you can install your software"
— C. Titus Brown, "Virtual machines considered harmful for reproducibility"
In [1]:
!conda install -y conda-build -q -n root
conda packages are created from conda recipes. We can create a bare recipe using conda skeleton to build it from a PyPI package.
In [3]:
!conda skeleton pypi pytest-benchmark > /dev/null
In [4]:
!ls pytest-benchmark
These are the minimum files for the recipe:
meta.yaml contains all the metadatabuild.sh and bld.bat are the build scripts for Linux/OS X and Windows respectivelymeta.yaml fileIt contains the metadata in YAML format.
package, source and build specify the name, version and source of the packagerequirements specify the build (install time) and run (runtime) requirementstest specify imports, commands and scripts to testabout adds some additional data for the package
In [26]:
!grep -v "#" pytest-benchmark/meta.yaml | head -n24
In [28]:
!cat pytest-benchmark/build.sh
In [30]:
!grep -v "::" pytest-benchmark/bld.bat
Adapted from http://conda.pydata.org/docs/building/recipe.html#conda-recipe-files-overview
Seems legit!
In [32]:
!conda build pytest-benchmark --python 3.5 > /dev/null # It works!
In [33]:
!ls ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2
In [35]:
!conda install pytest-benchmark --use-local --yes
Let's upload the package first using anaconda-client:
In [38]:
!conda install anaconda-client --quiet --yes
In [39]:
!anaconda upload ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2
And now, let's install it!
In [47]:
!conda remove pytest-benchmark --yes > /dev/null
In [48]:
!conda install pytest-benchmark --channel juanlu001 --yes
In [3]:
!conda convert ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 --platform all | grep Converting
python as a build or run dependency!If the recipe builds on a fresh, headless, old Linux it will work everywhere
conda-forge is a github organization containing repositories of conda recipes. Thanks to some awesome continuous integration providers (AppVeyor, CircleCI and TravisCI), each repository, also known as a feedstock, automatically builds its own recipe in a clean and repeatable way on Windows, Linux and OSX.
Features:
What I love:
meta.yaml, the only option is to keep a copy of all dependenciesThe state of Python packaging is improving upstream too!
Still, there are some remaining irks: